Skip to content

feat(pytorch): add PyTorch 2.13 currency scaffold#6365

Open
bhanutejagk wants to merge 11 commits into
mainfrom
pytorch-2.13-currency
Open

feat(pytorch): add PyTorch 2.13 currency scaffold#6365
bhanutejagk wants to merge 11 commits into
mainfrom
pytorch-2.13-currency

Conversation

@bhanutejagk

Copy link
Copy Markdown
Contributor

Summary

Scaffold for the PyTorch 2.13 currency release on the V2 pipeline. Adds DLC image configs, workflow callers, per-version pyproject/uv locks, and release-schedule entries for PyTorch 2.13.0 on cu130 (Amazon Linux 2023), covering ec2 and sagemaker customer types across cuda and cpu device types.

No shared Dockerfile changesdocker/pytorch/Dockerfile.{cuda,cpu} remain shared across all versions; per-version pins live under docker/pytorch/2.13/{cuda,cpu}/.

Version pins

Package Pin Source
torch 2.13.0 https://github.com/pytorch/pytorch/releases/tag/v2.13.0
torchvision 0.28.0 https://github.com/pytorch/vision/releases/tag/v0.28.0 (release/0.28)
torchaudio 2.11.0 (held) Upstream pytorch/audio has no 2.12 or 2.13 release tag; latest is v2.11.0. Matches the 2.12 DLC pin.
CUDA 13.0.2 (cu130) Only branch on download.pytorch.org/whl/* with a complete torch + torchvision + torchaudio wheel set for the pinned versions (cu132 has torch+vision but no torchaudio).
Python 3.12 Unchanged from 2.12.
NCCL / EFA / GDRCopy / flash-attn / TE / DeepSpeed Unchanged from 2.12 Minor bump only.

uv lock re-resolved cleanly in both 2.13/cuda and 2.13/cpu.

Files added

  • docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock} (4 files)
  • .github/config/image/pytorch/2.13-{ec2,sagemaker}-{cuda,cpu}.yml (4 configs)
  • .github/workflows/pytorch.{pr-2.13-cuda,pr-2.13-cpu,autorelease-2.13-ec2,autorelease-2.13-sagemaker}.yml (4 callers)
  • .github/release-schedule.yml — 2 new entries (Tue/Thu, 30-min grid, light GPU class)

Autorelease cron slots

Placed on Tue/Thu inside the documented 16:00–23:00 UTC release window, on the existing 00/30-minute grid, with a free 30-min buffer around each pick:

Workflow Cron Immediate neighbors
pytorch.autorelease-2.13-ec2.yml 30 19 * * 2,4 19:00 vllm-omni-ec2 (medium); 20:00 sglang-ec2-ubuntu (heavy)
pytorch.autorelease-2.13-sagemaker.yml 00 23 * * 2,4 22:00 vllm-omni-sagemaker (medium); 22:30 openfold3-sagemaker (heavy, Tue only)

Both entries in release-schedule.yml match the on: schedule: blocks inside the workflow files (_prcheck.release-schedule.yml should pass).

Test plan

  • pytorch.pr-2.13-cuda.yml triggers on this PR (paths match .github/config/image/pytorch/2.13-*-cuda.yml, docker/pytorch/2.13/cuda/**) — build + sanity + security + single-gpu + efa + telemetry + unit + sagemaker
  • pytorch.pr-2.13-cpu.yml triggers on this PR — build + sanity + security + telemetry + unit
  • Style/prcheck workflows pass (_prcheck.release-schedule.yml validates the cron correspondence)
  • Autorelease workflows do NOT fire on the PR (they're schedule + workflow_dispatch only); they will fire on the first Tue/Thu after merge
  • Manual workflow_dispatch smoke of pytorch.autorelease-2.13-ec2.yml after merge to confirm the release path

Add DLC image configs, workflow callers, and release schedule entries
for PyTorch 2.13.0 on cu130 (Amazon Linux 2023). Covers ec2 and
sagemaker variants across cuda and cpu devices.

- docker/pytorch/2.13/{cuda,cpu}/{pyproject.toml,uv.lock} pinned to
  torch 2.13.0, torchvision 0.28.0, torchaudio 2.11.0 (held; upstream
  audio has no 2.12/2.13 tag yet)
- 4 image configs and 4 workflow callers (pr + autorelease x ec2/sagemaker)
- autorelease crons: 30 19 UTC ec2, 00 23 UTC sagemaker on Tue/Thu
Align 2.13 currency image with newer upstream releases that were
available when scaffolding but held back from the initial 2.12-copy.

- nccl 2.26.2 -> 2.30.7-1: matches torch 2.13's bundled NCCL pin
  (torch 2.13 requires source builds to use NCCL >= 2.23 and pins its
  bundled runtime NCCL to 2.30)
- deepspeed 0.18.8 -> 0.19.2: 0.19.x train carries the torch >= 2.11
  compatibility patches; 0.18.8 predates them
- transformer_engine 2.12.0 -> 2.16.1: security patch + cuDNN 9.18.1
  requirement in the latest stable train (2.12 is 4 months old)
- efa 1.47.0 -> 1.49.0: latest EFA installer (libfabric 2.4.0amzn5.0,
  efa driver 3.1.0, OFI NCCL Plugin 1.20.0)
- gdrcopy 2.4.4 -> 2.6: latest stable; new optional DMA-BUF backend
  requires cuda driver 13.3+ but falls back to gdrdrv (current path)
- pyproject deepspeed==0.18.8 -> 0.19.2 mirrors the config bump
- uv.lock regenerated in 2.13/{cuda,cpu}

cryptography>=49.0.0 bump was attempted but reverted: mlflow>=3.9.0
caps cryptography<49 in the sagemaker extra. Revisit when mlflow's
upper cap loosens.

CPU configs unchanged (no gpu-only pins to bump).
@bhanutejagk bhanutejagk marked this pull request as ready for review July 9, 2026 00:01
The wheel-cache upload path in
scripts/ci/build/pytorch_runtime/lib/upload_wheels.sh runs
`docker buildx build --target wheel-export`, which re-executes the
entire Dockerfile to reach a scratch stage that COPYs the built wheels
out of the flash-attn and TE builder stages. With MAX_JOBS=8, the
concurrent flash-attn 2.8.3 + transformer-engine 2.16.1 source builds
during that re-execution exceed the CI runner cgroup memory limit and
get OOM-killed (SIGKILL / exit code 137, PR #6365 attempt 2).

Halving MAX_JOBS to 4 roughly halves peak compile RAM, keeping the
re-execution under the runner ceiling at the cost of a slower flash-attn
/ TE compile step. CPU configs unaffected.
@bhanutejagk bhanutejagk force-pushed the pytorch-2.13-currency branch from 51e3d4c to eb84c58 Compare July 9, 2026 18:06
Bhanu Teja Goshikonda added 8 commits July 9, 2026 13:41
Revert the temporary max_jobs=8->4 workaround (eb84c58). Root cause of
the wheel-export OOM is x86-build-runner is CPU-sized and can't handle
the concurrent flash-attn + TE source recompile that
`docker buildx build --target wheel-export` triggers.

Route pytorch GPU builds (device_type=gpu) to x86-vllm-build-runner,
the existing GPU-compilation fleet already used by vllm and vllm-omni.
CPU builds stay on x86-build-runner. No new CDK / infra changes needed;
the fleet is already provisioned with GPU-compilation-appropriate RAM.
The dedicated x86-pytorch-build-runner fleet is defined in
DLContainersInfraCDK mainline (commit 85a06e065, rebalanced by
f4bcbc9c on 2026-06-30) - identical hardware to x86-vllm-build-runner
and x86-sglang-build-runner, but reserved for pytorch.

Route all pytorch builds (both CPU and GPU) to the dedicated fleet
unconditionally, matching the vllm.pipeline.yml and sglang.pipeline.yml
pattern. This replaces the temporary conditional that routed GPU builds
to x86-vllm-build-runner as a stopgap.

Cache locality on a dedicated fleet is expected to fix the wheel-export
OOM: BuildKit finds the primary build's flash-attn + TE layers still
cached on the same host, so the wheel-export re-run reduces to a
COPY --from=<cached-layer> instead of a concurrent recompile.

- pytorch.pipeline.yml: unconditional fleet:x86-pytorch-build-runner
- ci-architecture.md: document the new fleet row
PR-6365 ECR vulnerability scan flagged 2 HIGH CVEs on 2.13 CPU
images (also on SageMaker CPU variant). Both are unbounded-memory /
catastrophic-backtracking bugs in soupsieve's CSS selector parser,
fixed upstream in 2.8.4.

soupsieve is transitive via beautifulsoup4 -> python-fasthtml ->
fastai. Add an explicit direct pin `soupsieve>=2.8.4` to both cpu
and cuda pyproject.toml (following prior transitive-CVE bump
convention: msgpack, tornado, cryptography, aiohttp), regenerate
uv.lock via `uv lock`.

CVEs cleared:
- GHSA-2wc2-fm75-p42x (unbounded memory on large selector lists)
- GHSA-836r-79rf-4m37 (regex catastrophic backtracking on attribute selectors)

Upstream fix pin: soupsieve>=2.8.4
The wheel-cache upload path re-runs `docker buildx build --target
wheel-export` to extract compiled wheels from the wheel-export stage.
Because that re-invocation passed NO --build-arg while the primary
build passes ~15 of them, BuildKit computed a different cache key for
every stage and had to recompile flash-attn + TE from source
concurrently, OOM-killing the CodeBuild runner (exit 137 at
upload_wheels.sh:45).

Forward the same --build-args the primary build uses (read from
EXTRA_BUILD_ARGS env var, the space-separated list of arg names the
build-image action already computes via resolve_build_args.py). The
wheel-export stage now hits BuildKit's local cache for every prior
stage and reduces to a COPY --from=<cached-layer>, avoiding the
recompile entirely.
Reverts commit 2705344. That change forwarded primary-build --build-args
to the wheel-export re-invocation to make BuildKit cache-hit and avoid
recompiling flash-attn + TE.

No longer needed: the x86-pytorch-build-runner fleet has been bumped from
m6a.8xlarge (128 GB) to m6a.12xlarge (192 GB) via DLContainersInfraCDK CR
CR-288126860. The runner now has enough memory headroom to complete the
cold-cache wheel-export recompile without OOM.

Reverting keeps upload_wheels.sh in its upstream shape for other
frameworks that share the pattern (though currently only pytorch_runtime
uses this exact copy).
Starting with EFA 1.48, the installer auto-detects NGC containers via
/opt/nvidia/nvidia_entrypoint.sh (present in nvidia/cuda:*-amzn2023 base
images) and skips installing the AL2023 libnccl-ofi RPM. This left the
pytorch runtime image without libnccl-net.so, breaking every unit /
sanity / single-gpu / efa test that asserts the OFI plugin is present.

The sibling scripts/docker/common/install_efa_amzn2023.sh already has
this fix (lines 37-48); mirror the version-gated --disable-ngc flag here
so aws-ofi-nccl installs regardless of the base-image NGC detection.

Only affects pytorch - sglang, vllm, and vllm-omni use the common
installer via their own paths.
Both fetch_wheels.sh and upload_wheels.sh keyed the S3 wheel cache
by ${PKG}/cu${MAJOR}${MINOR}/${FNAME}, omitting the torch version the
wheel was compiled against. Since pytorch 2.12 and 2.13 both pin
flash_attn 2.8.3 + cuda 13.0.2, the 2.13 build fetched 2.12's cached
wheel and loaded it into torch 2.13 - producing an undefined symbol
at import (c10::impl::cow::materialize_cow_storage, removed in
torch 2.13).

Key by torch major.minor as well, so cross-torch-version cache reuse
is impossible by construction. transformer-engine-torch would hit the
same bug at the next torch bump.

Note: stale cu130/torch212 wheels remain in s3://dlc-cicd-wheels/
under the old key format. They're now orphaned and unreferenced -
safe to delete via a separate S3 cleanup, or leave as is.
TE 2.16.1 calls cublasLtGroupedMatrixLayoutInit_internal which requires
libcublasLt.so.13.3, but our runtime base image (nvidia/cuda:13.0.2-
runtime-amzn2023) ships an older cuBLASLt. The build succeeds against
devel headers but fails to load at runtime with:

  ImportError: /path/libtransformer_engine.so: undefined symbol
  cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13

TE 2.13.0 predates the cuBLAS 13.3 dependency (grouped GEMM with
on-device group sizes landed in TE 2.14 via NVIDIA/TransformerEngine
#2748) and is torch-2.13 compatible. Bump base CUDA to 13.3.x-amzn2023
in a follow-up once that image is available on Docker Hub.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant